BOUNDEDNESS FOR PSEUDO-DIFFERENTIAL 
CALCULUS ON NILPOTENT LIE GROUPS 
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Abstract. We survey a few results on the boundedness of operators 
O-l arising from the Weyl-Pedersen calculus associated with irreducible rep- 

resentations of nilpotent Lie groups. 



1. Introduction 

Convolution operators and global Weyl calculus on nilpotent Lie groups 
have been extensively studied in many papers, in connection with various 
problems in partial differential equation and representation theory (see |10t 
[El [la [II El IMl 1211 [in |23l [26]). It has been repeatedly remarked that 
the global Weyl calculus is an extension of the classical Weyl calculus on 
R"; however, to see this, some further identifications and results are needed 
(see [22]). This phenomenon has roots in the fact that the global Weyl 
calculus is not injective, and thought it is associated to a given irreducible 
representation, the link with the corresponding coadjoint orbit is not clear. 
These issues were resolved by N.V. Pedersen in f25] , who constructed a Weyl 
calculus — that we call Weyl-Pedersen calculus — associated to an irreducible 
representation of a nilpotent Lie group, which is a bijection between good 
function spaces of symbols defined on the corresponding orbit and operators 
defined on the Hilbert space of the representation. In addition, this calculus 
directly extends the classical Weyl calculus. 

The aim of this paper is to survey some boundedness results for the Weyl- 
Pedersen calculus in the case of flat orbits and to give further applications to 
some three-step nilpotent Lie groups that have non-flat generic orbits. The 
results are generalizations of the classical C alder on- Vaillancourt theorem [9] 
and of the Beals characterization of the pseudo-differential operators [3] (see 
also [8]). 

Main definitions are given in Section [21 along with an example illustrating 
the non-injectivity of the global Weyl calculus. The boundedness results are 
given in Section [S] 
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Finally, let us mention here that other extensions of the classical Weyl 
calculus have been constructed in terms of representations, for instance the 
magnetic calculus on M"' [19], and nilpotent Lie groups [5], Weyl calculus on 
nilpotent p-adic Lie groups [15], [4]. 

A good source for the background information on nilpotent Lie groups 
and their representations is [llj . 

2. Weyl calculi for representations of nilpotent Lie groups 

Preliminaries on nilpotent Lie groups. Throughout this paper the 
nilpotent Lie groups are supposed to be connected and simply connected. 
Therefore there is no loss of generality in assuming that a nilpotent Lie 
group is a pair G = (g, •), where g is a nilpotent Lie algebra (over M unless 
otherwise mentioned) with the Lie bracket [•,■], and the group multiplication 
• is given by the Baker-Campbell-Hausdorff series: 

1 1 

(Vx,?/ Gfl) x-y = x + y + -[x,y\ + -^{[x, [x,y\] + [y, [y,x\]) H 

If the Lie algebra q is nilpotent of step n, the group multiplication g x g — )• g, 
(x, y) 1-^ X • y is a polynomial mapping of degree n. With this identification 
the exponential from the Lie algebra to the group is then the identity, while 
the inverse of x € g is — x, and the unit element is € g. 

We recall that for every ^ € g* the corresponding coadjoint orbit is 



where := {x G g | (Ad^x).^ = ^} is the coadjoint isotropy group (or the 
stabilizer of ^), and Ad^: G x g* ^ g* stands for the coadjoint action. We 
use the notation g^ for the corresponding Lie algebra, called the radical of 



We will always denote by dx a fixed Lebesgue measure on g and we recall 
that this is also a two-sided Haar measure on the group G. We let d^ be 
a Lebesgue measure on g* with the property that if we define the Fourier 
transform for every a € L^(g*) by 



then we get a unitary operator F: L^(g*,d^) — > -L^(g,dx). We also de- 
note by 5(g) ( 5(g*)) the Schwartz space on g (respectively g*), by 4S'(g) 
(<S'(g*))) its topological dual consisting of the tempered distributions, and 
by (•,•): 5'(g) x 5(g) — )■ C the corresponding duality pairing. The Fourier 
transform extends to a linear topological isomorphism 5'(g*) — t- 5'(g). 

We recall that according to a theorem of Kirillov (see [11^ Chap 2] ) , there 
exists a natural one-to-one correspondence between the coadjoint orbits and 
the equivalence classes of unitary irreducible representations of a nilpotent, 
connected and simply connected Lie group G. 



:= {(Ad^x)e I X G g} ~ G/G^ 




0* 
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Example 2.1. For every integer n > 1 let (• | ■) denote the Euclidean scalar 
product on M". The Heisenberg algebra is {)2n+i = M" x M" x M with the 
bracket [{q,p,t), {q',p',t')] = [(0,0, {p \ q') — {p' \ q))]. The Heisenberg group 
is ]H[2n+i = (f)2n+i, ') with the multiplication x ■ y = x + y + ^[x,y]. If we 
identify the dual space f)2n+i with x R" x R in the usual way, then every 
coadjoint orbit belongs to one of the following families: 

i) the affine hyperplanes Or = R" x R" x {l/h} with e R \ {0}; 

ii) the singletons Oa,b = {(a, &, 0)} with a, 6 G R". 

For every € R \ {0} there is a unitary irreducible representation on the 
Hilbert space L2(R"), namely vr?^: El2„+i ^ B{L'^{W)) defined by 

{Trn{q,p,t)f){x) = e'«Pl^')+5(pl'?)+|)/(g + 2;) for a.e. x G R" (2.1) 

for arbitrary / G L^(R") and {q,p,t) G El2n+i- This is the Schrddinger rep- 
resentation of the Heisenberg group Bl2n+i, and corresponds to the coadjoint 
orbit Oh in the first family above. Moreover, for every a,b £ R" there is a 
unitary irreducible representation on a 1-dimensional Hilbert space, namely 
T^{a,b) ■ El2n+i U (1) := {z € C \ \z\ = 1} defined by 

vr(.,6)(g,P,t) = e'«^'l")+(''l''» 

for all {q,p,t) G ]H2n+i, and corresponds to the orbit Oafi in the second 
family. 

Weyl calculi on nilpotent Lie groups. 

Definition 2.2 ([1], [2]). Let G = {g, ■) be a nilpotent Lie group. If vr: g — > 
B{'H) is a unitary representation of G, then we can use Bochner integrals 
for extending it to a linear mapping 

tt: L^{g) Bin), 7r{b)v = j b{x)-K{x)vdx if 6 G L^{q) and v eU. 

3 

Let 7~ be a locally convex space which is continuously and densely em- 
bedded in and has the property that for every h,x ^ T we have (vr(-)/i | 
x) G 5(g). Then we can further extend vr to a mapping 

tt: S'{q) ^ CiT,T*), {<u)h I x) = {n, {7r{-)h \ x)) 

Hue S'{q), h eT, and x^T- 

where T* is the strong antidual of T, and (• | •) denotes the anti-duality 
between T and T* that extends the scalar product of %. Here we have used 
the notation C{T,'T*) for the space of continuous linear operators between 
the above spaces (these operators are thought of as possibly unbounded 
linear operators in H). 

In the same setting, the global Weyl calculus for the representation tt is 
the mapping 

OP: T-^L^{g) ^ B{n), OP(a) = 7r(J^a) 
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which further extends to 

OP: S'{q*) ^ C{r,r*), OP:=TToT-\ 

Example 2.3. Consider the (left) regular representation 

A: g ^ B{L\5)), (Hx^y) = 0((-x) • y). 

If we extend it as above to A : L^{q) — >■ B{L'^{q)) then we obtain for b € L^is) 
and ip G L'^id) 

{X{b)i^){y)= I b{x)i^{{-x)-y)dx={b*tP){y) 


hence we recover the convolution product, which makes L^{q) into a Banach 
algebra. 

The above construction of the global Weyl calculus for the regular repre- 
sentation is usually considered with T = S{g), yielding 

0P:5'(B*)^/:(5(g),5'(g)) 

and the related mapping 

A:5'(g)^£(5(0),5'(g)) 

whose values are the (possibly unbounded) convolution operators on the 
nilpotent Lie group G. 

In the case of the global Weyl calculus the symbol of an operator in the 
representation space of an irreducible representation may not be uniquely de- 
termined on the corresponding coadjoint orbit, unlike in the case of pseudo- 
differential Weyl calculus on M" or two step nilpotent groups. (See Ex- 
ample 12.71 and also |23j . ) The Kirillov character formula says that when 
a € 5(g*), and vr: G i— )• B{T-l) is a irreducible representation, then OP(a) 
is a trace class operator and there exists a constant that depends on the 
unitary class of equivalence of vr only, such that 

Tr(OP(a)) =C I a(e)de, 
o 

where O is the coadjoint orbit corresponding to vr. This seems to suggest 
that OP(a), when a £ C°°(g) n5'(g), depends only on the restriction of a to 
the coadjoint orbit O. This is not always the case, as it could be seen from 
the next example [20l App. Sect. I]; see also [24, Ex. N4N1, pag. 9-10]. 

Example 2.4. Let g be the 4-dimensional threadlike (or filiform) Lie alge- 
bra. Equivalently, g is 3-step nilpotent, 4-dimensional, and its center is 
1-dimensional. Then g has a Jordan-Holder basis {Xi, X2, X^, X4} satisfy- 
ing the commutation relations [X4,X3] = [^4,^2] = Xi and [Xj,^^] = if 
1 < A; < J < 4 with (j, k) ^ {(4, 3), (4, 2)}. Let O be the coadjoint orbit of 
the functional 



^0 : g ^ M, CoihXi + t2X2 + t3^3 + t4^4) = ^l- 
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Then dimO = 2, and if we identify g* to by using the basis dual to 
{Xi,X2,X3,X4}, then we have O = | s,t G M} C M^. A uni- 

tary irreducible representation tt: g ^ S(L^(M)) associated with the coad- 
joint orbit O can be defined by 



where we denote by t both the variable of the functions in L^(M) and the 
operator of multiplication by this variable in L^(M). 

It is clear that the function a: — )■ M, 0(2/1,^2,2/3,2/4) = yiC^ys — 2/1)) 
vanishes on the coadjoint orbit O, and on the other hand it was noted in 
[201 pag. 236] that OP''(a) = 

N.V. Pedersen introduced in |25j an orbital Weyl calculus that is specific 
to a given orbit, or equivalently, to a class of unitary irreducible represen- 
tations and that, in addition, gives isomorphism between Schwartz symbols 
defined on the orbit and regularizing operators defined in the space of the 
representation. The calculus may depend on the choice of a Jordan-Holder 
basis. 

To describe this Weyl-Pedersen calculus we need first some notation. Let 
G = (g, •) be a nilpotent Lie group of dimension m > 1 and assume that 
{Xi, . . . , Xm} is a Jordan-Holder basis in g; so for j = 1, . . . , m if we define 
gj := span{Xi, . . . ,Xj} then [Q,Qj] C where go := {0}. Let vr: g 

3(1-1) be a unitary representation of G associated with a coadjoint orbit 
O C g*. Pick ,^0 £ O, denote e := {j \ Xj Qj^i + g^g}, and then define 
ge := spanjXj \ j £ e}. We have g = g^g + 0e and the mapping O ^ g*, 
C ^ Clse' is a diffeomorphism. Hence we can define an orbital Fourier 
transform S'{0) 5'(ge), a >—^a which is a linear topological isomorphism 
and such that for every a G S{0) we have 



Here we have the Lebesgue measure dx on corresponding to the basis 
{Xj I j G e} and d^ is the Borel measure on O such that the aforementioned 
diffeomorphism O — )• g* is a measure preserving mapping and the Fourier 
transform L'^{0) — t- L^(ge) is unitary. The inverse of this orbital Fourier 
transform is denoted by a 1— t- a. 

Definition 2.5 (|25j). With the notation above, the Weyl-Pedersen calculus 
associated to the unitary irreducible representation vr is the mapping 



The space of smooth vectors l-Loo := {f G ?^ | 7r(-)f G C°°(g,?^)} is dense in 
l-L and has the natural topology of a nuclear Frechet space with the space 
of the antilinear functional denoted by T-L-oo '■= T~i*oo (with the strong dual 



d7r(Xi) = il 



d^(X2) = -it, d7r(X3) = y, d7r(X3) 



dt 




o 
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topology). One can show that the Weyl-Pedersen calculus extends to a linear 
bijective mapping 

Op, : S'{0) ^ C{noo,n-oo), {Op^{a)v I w) = (a, (vrO | w)) 

for a € S'{0), v,w ^ T-Coo, where in the left-hand side (• | •) denotes the ex- 
tension of the scalar product of Ti to the sesquilinear duality pairing between 
Tioo and H-oo- 

Note that in fact Op, is associated to the coadjoint orbit corresponding 
to the representation tt. Indeed if vri and tt are unitary equivalent represen- 
tations, Op, and Op,^ are defined on the same space of symbols, there is 
a unitary operator U such that Op,^(a) = U*Op^{a)U for every a G S{0), 
and this equality extends naturally to a G S'{0). Therefore, whenever the 
orbit O is fixed and no confussion may arise, we use the notation Op instead 
of Op,, for any irreducible representation corresponding to O. 

Remark 2.6. The map 5(g*) — )■ /^("H-oo, ^00)1 a ^ OP(a) is surjective [18j, 
while S{0) — /^('H-ooi ^oo), a 1— )• Op,(a) is bijective [25]. In fact, it follows 
by |25i Thm.2.2.7] that OP(a) = Op,(6), where 

b{x) = Co,eTV(^(-x)OP(a)), X G Qe, 

and Co,e is a constant that depends on the Jordan-Holder basis and on the 
coadjoint orbit O. 

Example 2.7. The Weyl-Pedersen calculus for the Schrodinger representation 
of the Heisenberg group IHl2n+i is just the usual Weyl calculus from the 
theory of partial differential equations on M", as developed for instance in 
[IS] and [IT]. In this case we have Ti = L'^{W) and 'Hoo = 5(M"). 

3. BOUNDEDNESS FOR THE ORBITAL WeYL CALCULUS 

Let G = (0, •) be a nilpotent Lie group with a unitary irreducible rep- 
resentation tt: Q —7- B{T-L) associated with the coadjoint orbit O Q q*. 
One proved in [6J that if the symbols belong to suitable modulation spaces 
^00,1 ^ S'{0), then the corresponding values of the Weyl-Pedersen calcu- 
lus belong to B{7i). This condition does not require smoothness properties 
of symbols. 

We will now describe a result established in [7J which extends both the 
classical Calderon-Vaillancourt theorem and Beals' criterion on the Weyl 
calculus, by linking growth properties of derivatives of smooth symbols to 
boundedness properties of the corresponding pseudo-differential operators 
in the case when the coadjoint orbit O is flat, in the sense that for some 
^0 € O we have O = € Q* \ = Co\^}, where 3 is the center of g. 
This is equivalent to the condition dimC = dimg — dim 3, and it is also 
equivalent to the fact that the representation vr is square integrable modulo 
the center of G. Since this hypothesis on coadjoint orbits might look more 
restrictive than it really is, we recall that every nilpotent Lie group embeds 
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into a nilpotent Lie group whose generic coadjoint orbits (that is, the ones 
of maximal dimension) are flat [11^ Ex. 4.5.14]. 

If O is a generic flat coadjoint orbit, then the Weyl-Pedersen calculus 
Op: s'{o) ^ c{noc,n _oo) is a linear topological isomorphism which is 
uniquely determined by the condition that for every b € S{g) we have 

Op((J'~^6)Io) = j Tr{x)b{x)dx, 

B 

(see [251 Th. 4.2.1]). 

We define Diff (O) as the space of all linear differential operators D on O 
which are invariant to the coadjoint action, in the sense that 

(Vx G 0)(Va e C^{0)) D{a o Ad^(x)lo) = (Da) o Ad^(x)|o. 

Let us consider the Frechet space of symbols 

C~(0) = {a G C7°°(0) 1 Da G L°°(0) for ah D G Diff (O)}, (3.1) 

with the topology given by the seminorms {a i-)- Il-Da]|ioo(c))}£)gj3ifj(0). 

Theorem 3.1 ([Z]). Let G be a connected, simply connected, nilpotent Lie 
group whose generic coadjoint orbits are flat. Let O be such an orbit with 
a corresponding unitary irreducible representation n: G ^ B{'H). Then for 
a G C°°(0) we have 

a G e^{0) ^ (VD G Diff (O)) Op(Da) G B{n). 

Moreover the Weyl-Pedersen calculus defines a continuous linear map 

and the Frechet topology ofC^{0) is equivalent to that defined by the family 
of seminorms {a ^ \\Op{Da)\\}D(zDm {o)- 

Let C^{0) be the space of all a G C°°{0) such that the function Da 
vanishes at infinity on O, for every D G Diff (O). It easily follows by the 
above theorem that if a G C°°{0) then 

a G C^iO) (VD G Diff (O)) Op{Da) is a compact operator. 

If vr is the Schrodinger representation of the (2n + l)-dimensional Heisen- 
berg group, then Theorem 13. II characterizes the symbols of type Sqq for the 
pseudo-differential Weyl calculus Op: ^'(M^") ^ £(5(M"), 5'(]R") j. Name- 
ly, for any symbol a G C°°(]R^") we have 

(Va G N2") G L°°(M2") ^ (Va g N^") Op(a"a) G ^(^^(IR")), 

where d°' stand as usually for the partial derivatives. The above statement 
unifies the Calderon-Vaillancourt theorem and the so-called Beals' criterion 
for the Weyl calculus. 
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Application to 3-step nilpotent Lie groups. 

Proposition 3.2. Let G = (3, •) he a nilpotent Lie group with an irreducible 
representation vr: g — ?> B{'H) associated with the coadjoint orbit O C g*. // 
H = (f), •) is a normal subgroup of G, then the following assertions are 
equivalent: 

(1) The restricted representation -7r|(, : \] — )• B{H) is irreducible. 

(2) The mapping g* — )■ f)*, ^ 1— )■ ^|(, gives a dijjeomorphism of O onto a 
coadjoint orbit of H , which will be denoted by 0\\^. 

If this is the case, then the irreducible representation 7r|(, is associated with 
the coadjoint orbit of H . 

Proof. Pick any Jordan-Holder sequence that contains f). Then the impUca- 
tion[TJ =^{2J follows at once by iterating jlll Th. 2.5.1], while the converse 
implication follows by using Vergne polarizations. The details of the proof 
will be given elsewhere. □ 

Remark 3.3. In the setting of Proposition 13.21 if Xi, . . . ,Xm is a Jordan- 
Holder basis in g such that Xi , . . . , is a Jordan- Holder basis in [} for 
k = dimf), then the following assertions hold true: 

• The coadjoint G-orbit O and the coadjoint ff-orbit have the 
same set of jump indices e C {1, . . . , k}. In particular ge = f)e ^ 
and O cannot be flat. 

• We have the i?-equivariant diffeomorphism 0: O— 7'0|(,,.^M',^|(,. It 
intertwines the orbital Fourier transforms S'{0) — )• S'{Qe) = 5'(f)e) 
and S'{0\^) — > S'{t)e). Therefore, by using also the previous remark, 
we obtain 

(VaG5'(0)) Opja) = 0p,|^(aoe'i). (3.2) 

• The ff-equivariant diffeomorphism also induces a unital injective 
homomorphism of associative algebras 

Difr(O) Diff (0|f,), D^D\ 

such that D(ao0) = {D^a)oe for all a G C°°(0|^) and D G Diff (O). 
In particular, it follows that for p G {0} U [1, 00] we get a continuous 
injective linear map 

C°°'P{0)^C°°'P{0\i^), a^ao0-i (3.3) 

where we use the Frechet spaces 0°°'^ introduced in [71 Th. 4.4]. 

In particular. Proposition [3]2l ()3.2p and ()3.3p . along with Theorem 13.11 prove 
the next corollary. 

Corollary 3.4. If one of the equivalent conditions of Proposition [Ql holds 
true and if the orbit is fiat, then for every a G C^(0) we have Op^(a) G 
B{'H), and moreover the Weyl-Pedersen calculus defines a continuous linear 
map Op„: C^{0)^B{n). 
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Note that the coadjoint orbit of the representation vr in the above Corol- 
lary l3.4l may not be flat, and yet we have proved an L^-boundedness assertion 
just hke the one of Theorem 13. 1[ We now provide a specific example of a 
3-step nilpotent Lie group taken from [26], which illustrates this result. 

Example 3.5. Recall from Example l2. li the Heisenberg algebra f)2n+i = 1^^" x 
M with the Lie bracket given by [{x,t), {x',t')] = (0,a;(x,x')) for x,x' G M^"" 
and t,t' € M, where to: M^" x R^" — > M is the symplectic form given by 
^{iq,p),iQ',P')) = (P I q') - (P' I q) for iq,p),iq',p') G M" x M" = M^n. it 
easily follows that if we consider the symplectic group 

Sp(M2'^, w) = {T e M2„(M) I (Vx, x' G R^") co{Tx, Tx') = a;(x, x')} 

then every T G Sp(]R^",a;) gives rise to an automorphism ay G Aut(f)2n+i) 
by the formula axix^t) = {Tx,t) for all x G M^" and t G M. Moreover, the 
correspondence a: Sp(M^",u;) — > Aut(P)2n+i) — Aut(El2„4.i), T i-> ar, is a 
group homomorphism and is injective. On the other hand, if we write the 
elements of Sp(M^", lj) as 2 x 2-block matrices with respect to the decompo- 
sition M^" = M" X M", then it is well known that Sp(M^",u;) is a Lie group 
whose Lie algebra is 

sp(M2",a;) = |(^^ \A,B = B^,C = C^ GMniR)] 

where B^ stands for the transpose of a matrix B. In particular, 5p(M^",cj) 
has the following abelian Lie subalgebra 

SnW = { (° °) I C = G M„(M)} 

and the corresponding Lie subgroup of Sp(M^"',a;) is the abelian Lie group 

Snim = { (^^ fj\C = C^ e M„,(R)}. 

It is easily seen that the group S'„(M) acts (via a) on f)2n+i by unipotent 
automorphisms, and therefore the corresponding semidirect product G = 
iS'n(M) XfjE[2„+i is a nilpotent Lie group. Its Lie algebra is g = s„,(M) x i)2n+i- 
If we denote by sym„(M) the set of all symmetric matrices in M„(M) viewed 
as an abelian Lie algebra, then we have an isomorphism of Lie algebras 
Q ~ sym„(]R) X i}2n+i with the Lie bracket given by 

[iC,q,p,t), {C',q',p',t')] = (0,0, Cg' - C'q, (p \ q') - (/ | q)) 

for C,C" G sym„(R), q,q',p,p' G M" and t,t' G M. It easily follows by the 
above formula that [g, [g, [0,0]]] = {0} hence is a 3-step nilpotent Lie al- 
gebra. Moreover, f)2n+i ^ {0} x f)2n+i is an ideal of g and it was proved 
in |26j that for every coadjoint G-orbit O C g* of maximal dimension we 
have dimO = n and moreover the mapping ^ 1— >• C|i)2„+i gives a diffeomor- 
phism of O onto a coadjoint ]Hl2n+i-orbit 0|f)2„+i- Thus Proposition 13.21 and 
Remark 13.31 apply for the Weyl-Pedersen calculus of a unitary irreducible 
representation vr: g — ?■ BiJ-L) associated with the coadjoint orbit O. Note 
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that this result is similar to, and yet quite different from [26, Th. 8.6-8.7], 
inasmuch as we work here with an irreducible representation (see Defini- 
tion [23]) instead of the regular representation of G (see Example 12. Sh . 
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